Search for: All records

We study the game modification problem, where a benevolent game designer or a malevolent adversary modifies the reward function of a zero-sum Markov game so that a target deterministic or stochastic policy profile becomes the unique Markov perfect Nash equilibrium and has a value within a target range, in a way that minimizes the modification cost. We characterize the set of policy profiles that can be installed as the unique equilibrium of a game and establish sufficient and necessary conditions for successful installation. We propose an efficient algorithm that solves a convex optimization problem with linear constraints and then performs random perturbation to obtain a modification plan with a near-optimal cost. 

We study offline reinforcement learning (RL) with heavy-tailed reward distribution and data corruption: (i) Moving beyond subGaussian reward distribution, we allow the rewards to have infinite variances; (ii) We allow corruptions where an attacker can arbitrarily modify a small fraction of the rewards and transitions in the dataset. We first derive a sufficient optimality condition for generalized Pessimistic Value Iteration (PEVI), which allows various estimators with proper confidence bounds and can be applied to multiple learning settings. In order to handle the data corruption and heavy-tailed reward setting, we prove that the trimmed-mean estimation achieves the minimax optimal error rate for robust mean estimation under heavy-tailed distributions. In the PEVI algorithm, we plug in the trimmed mean estimation and the confidence bound to solve the robust offline RL problem. Standard analysis reveals that data corruption induces a bias term in the suboptimality gap, which gives the false impression that any data corruption prevents optimal policy learning. By using the optimality condition for the generalized PEVI, we show that as long as the bias term is less than the ``action gap'', the policy returned by PEVI achieves the optimal value given sufficient data. 

Abstract Ionospheric F‐region electron density is anomalously higher in the evening than during the daytime on many occasions in the summer in geomagnetic mid‐latitude regions. This unexpected ionospheric diurnal variation has been studied for several decades. The underlying processes have been suggested to be related to meridional winds, topside influx arising from sunset ionospheric collapse, and other factors. However, substantial controversies remain unresolved. Using a numerical model driven by the statistical topsideO⁺diffusive flux from the Millstone Hill incoherent scatter radar data, we provide new insight into the competing roles of topside diffusive flux, neutral winds, and electric fields in forming the evening density peak. Simulations indicate that while meridional winds, which turn equatorward before sunset, are essential to sustain the daytime ionization near dusk, the topside diffusive flux is critically important for the formation and timing of the summer evening density peak.